Update Kueue and Jobset controller default limit value #502

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merged

ycchenzheng merged 15 commits into AI-Hypercomputer:develop from ycchenzheng:chzheng/controller-manager-limit

Jul 10, 2025

Collaborator

ycchenzheng commented Jun 16, 2025 •

edited

Loading

Fixes / Features

Increase the memory limit of kueue-controller-manager and jobset-controller-manager to 1.2Mib per VM or 4Gi which is greater. The change was originally from b/421199006 and b/418017963. The change is implemented in utils

Testing / Documentation

Testing details.

[ y ] Tests pass
[ y ] Appropriate changes to documentation are included in the PR


          Update Kueue and Jobset controller default limit value

a875fdf

ycchenzheng requested review from Obliviour, 44past4, sharabiani, pawloch00, BluValor, gcie and RoshaniN as code owners

June 16, 2025 19:49

Collaborator Author

ycchenzheng commented Jun 16, 2025


          Update cluster.py

52915c9

RoshaniN reviewed

View reviewed changes

Collaborator

RoshaniN left a comment

Thanks Zheng!

Had couple of questions.

src/xpk/utils/kubectl.py Outdated Show resolved Hide resolved

src/xpk/utils/kubectl.py Outdated Show resolved Hide resolved

ycchenzheng requested a review from RoshaniN

June 17, 2025 01:32

SujeethJinesh reviewed

View reviewed changes

src/xpk/utils/kubectl.py Outdated Show resolved Hide resolved

SujeethJinesh reviewed

View reviewed changes

src/xpk/utils/kubectl.py Outdated Show resolved Hide resolved

pawloch00 reviewed

View reviewed changes

src/xpk/utils/kubectl.py Outdated Show resolved Hide resolved

pawloch00 reviewed

View reviewed changes

src/xpk/utils/kubectl.py Outdated Show resolved Hide resolved

pawloch00 reviewed

View reviewed changes

src/xpk/utils/kubectl.py Outdated Show resolved Hide resolved

pawloch00 reviewed

View reviewed changes

src/xpk/utils/kubectl.py Outdated Show resolved Hide resolved

pawloch00 reviewed

View reviewed changes

src/xpk/utils/kubectl.py Outdated Show resolved Hide resolved

pawloch00 reviewed

View reviewed changes

src/xpk/utils/kubectl.py Outdated Show resolved Hide resolved

pawloch00 reviewed

View reviewed changes

src/xpk/utils/kubectl.py Outdated Show resolved Hide resolved

pawloch00 reviewed

View reviewed changes

src/xpk/utils/kubectl.py Outdated Show resolved Hide resolved

ycchenzheng added 2 commits

June 23, 2025 15:28


          Split into get and update manifest

734a724


          Remove dup lines

f1a26d8

SujeethJinesh reviewed

View reviewed changes

src/xpk/commands/cluster.py Outdated Show resolved Hide resolved

src/xpk/utils/kubectl.py Outdated Show resolved Hide resolved

src/xpk/utils/kubectl.py Outdated Show resolved Hide resolved

src/xpk/utils/kubectl.py Outdated Show resolved Hide resolved

src/xpk/utils/kubectl.py Outdated Show resolved Hide resolved

src/xpk/utils/kubectl.py Outdated Show resolved Hide resolved

src/xpk/utils/kubectl.py Outdated Show resolved Hide resolved


          Organize code

66e263f

ycchenzheng commented

View reviewed changes

src/xpk/commands/cluster.py Outdated Show resolved Hide resolved

ycchenzheng added 3 commits

June 25, 2025 19:18


          Redesign the feature

389a62e


          Clean up code

d15cd93


          Correct wrong description

bab8a81

ycchenzheng commented

View reviewed changes

src/xpk/core/jobset.py Show resolved Hide resolved

ycchenzheng commented

View reviewed changes

src/xpk/core/kueue.py Show resolved Hide resolved

ycchenzheng added 2 commits

June 25, 2025 22:05


          Merge branch 'develop' into chzheng/controller-manager-limit


          Remove unnecessary section of yaml

c4719a9

ycchenzheng requested review from pawloch00, wstcliyu and SujeethJinesh

June 26, 2025 02:10

wstcliyu approved these changes

View reviewed changes

Collaborator

wstcliyu left a comment

LGTM. Please also reach out to XPK owners for verification. @Obliviour @pawloch00

Obliviour approved these changes

View reviewed changes

Collaborator

Obliviour left a comment

Thanks for this change, looking forward to avoiding limits as we scale. Can we confirm (if not already) with GKE team that these calculated values are good?

Collaborator Author

ycchenzheng commented Jul 1, 2025

@SujeethJinesh can you please take a look at this comment?

Collaborator

DannyLiCom commented Jul 3, 2025

@Obliviour Can you contact the GKE Team? Because we want to also create a tracking buganizer ticket for this issue/question, and then assign it to them.


          Merge branch 'develop' into chzheng/controller-manager-limit

ddebedf

SujeethJinesh approved these changes

View reviewed changes

Collaborator

SujeethJinesh commented Jul 7, 2025

@Obliviour These values are taken from our previous scale testing results. It's based directly on the number of VMs, and so this calculation is a small overestimate of what's needed. It should be enough.

SujeethJinesh added the release-bugfix label

ycchenzheng added 3 commits

July 8, 2025 01:31


          Resolve lint issue

7b8e64e


          Reformat the change

65c129c


          Merge branch 'develop' into chzheng/controller-manager-limit

1ca1fe4

Collaborator Author

ycchenzheng commented Jul 10, 2025

@pawloch00 This PR also has integration tests failing. Can you please add me to collaborator like #506 (comment) ?


          Merge branch 'develop' into chzheng/controller-manager-limit

0424d8a

Collaborator

pawloch00 commented Jul 10, 2025

@pawloch00 This PR also has integration tests failing. Can you please add me to collaborator like #506 (comment) ?

done

ycchenzheng mentioned this pull request

Update Kueue and Jobset controller default limit valueChzheng/controller manager update #526

Closed

ycchenzheng merged commit 8890e1a into AI-Hypercomputer:develop

42 of 46 checks passed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

SujeethJinesh SujeethJinesh approved these changes

Obliviour Obliviour approved these changes

wstcliyu wstcliyu approved these changes

44past4 Awaiting requested review from 44past4 44past4 is a code owner

sharabiani Awaiting requested review from sharabiani sharabiani is a code owner

BluValor Awaiting requested review from BluValor BluValor is a code owner

gcie Awaiting requested review from gcie gcie is a code owner

RoshaniN Awaiting requested review from RoshaniN RoshaniN is a code owner

pawloch00 Awaiting requested review from pawloch00 pawloch00 is a code owner

Labels